Skip to content

Refactor test modules and improve CLI error handling#91

Merged
alexkroman merged 11 commits into
mainfrom
claude/youthful-cori-vb7ovs
Jun 12, 2026
Merged

Refactor test modules and improve CLI error handling#91
alexkroman merged 11 commits into
mainfrom
claude/youthful-cori-vb7ovs

Conversation

@alexkroman

Copy link
Copy Markdown
Collaborator

Summary

This PR refactors test organization to keep modules under the 500-line gate, improves CLI error handling with better validation and suggestions, and enhances the onboarding wizard with JSON mode support and better error reporting.

Key Changes

Test Organization

  • Split test_init_command.py into test_init_force_and_report.py for --force semantics and port hints
  • Split test_login.py into test_login_guards.py for keyring preflight and offline whoami tests
  • Split test_onboard_sections.py into test_onboard_environment.py for environment section tests
  • Added test_streaming_diagnostics.py for WebSocket hygiene helpers
  • Moved environment-related test helpers to shared fixtures

CLI Error Handling & Validation

  • Init command: Added UsageError for file-exists conflicts (exit 2 instead of internal error); --force now warns when overwriting existing files
  • Transcribe command: Added client-side validation for --temperature (0-1 range) and negative --audio-end values
  • LLM command: Added validation for transcript IDs before network calls
  • Account command: Improved date parsing with _parse_day() helper and better error messages
  • Eval command: Added _failed_result() for graceful error handling in batch operations
  • Login command: Keyring usability check happens before browser flow or API key validation

Onboarding Wizard Improvements

  • Added JSON mode support to wizard sections with json_ctx fixture
  • Improved error reporting with _capture_stderr() helper
  • Better section result handling with _patch_sections() for testing
  • Enhanced environment section with doctor check status reporting

Voice Agent & Streaming

  • Refactored aai_cli/agent/voices.py to use Voice dataclass with language grouping
  • Added VOICE_NAMES constant for validation and ENGLISH/MULTILINGUAL language groups
  • Created aai_cli/streaming/diagnostics.py for shared WebSocket error handling and logger silencing
  • Improved handshake error classification (401/403) with actionable suggestions

Help Text & Output

  • Updated help text formatting in snapshots (line wrapping improvements)
  • Added --sandbox flag placement hints in examples (must come before subcommand)
  • Improved error messages with better grammar ("a"/"an" article selection)
  • JSON error mode emits proper {"error": …} envelopes on stderr

Configuration & Utilities

  • Added validate_profile() for profile name validation
  • Added validate_out_path() for early --out validation before transcription
  • Added consent_source() to telemetry for tracking consent decision layer
  • Improved whitespace-only env key handling in config.resolve()

Dev Server & Deployment

  • Added port change notification on stderr when port is busy
  • Improved --port handling with notify_port_change() warning
  • Better error handling in deploy confirmation flow

Auth Flow

  • Updated _open_browser() signature to accept **kwargs for flexibility
  • Added _note() helper for structured JSON/human progress notes during login
  • Improved browser flow error handling with better suggestions

Notable Implementation Details

  • Error classification: WebSocket handshake errors (401/403) are now distinguished from other failures with actionable next steps
  • Validation ordering: Client-side validation happens before network calls to fail fast and avoid unnecessary API charges
  • JSON mode consistency: All commands now emit proper error envelopes in JSON mode instead of mixing formats
  • Test fixtures: Shared helpers like _capture_stderr(), _patch_sections(), and _login_result() now accept json_mode parameter for consistent testing
  • Help text: Examples now show correct flag placement (global flags before subcommands)

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6

claude added 8 commits June 12, 2026 00:13
…tion

- client.py (_sdk_errors): compact httpx 'Request: <...>' reprs out of generic
  API error messages, strip the SDK's redundant 'failed to retrieve
  transcript(s):' preamble, and attach a network suggestion (parity with the
  llm gateway path).
- llm.py: only suggest a paid plan on gateway 401/403 when the response
  actually mentions the entitlement (plan/upgrade/billing/no access);
  otherwise point at the key/network so a proxy 403 doesn't send users
  to billing.
- llm command: validate --transcript-id locally before auth/network (same
  check as transcripts get), add min=1 to --max-tokens, warn on stderr when
  piped stdin is ignored because --transcript-id takes priority (quiet
  suppresses; --json gets a structured {"warning": ...} line), and reject
  -o/--output in --list-models mode (it was silently ignored).
- transcripts get: run the cheap local id validation before auth so a
  malformed id errors immediately instead of after login.
- sessions list: --status is now a closed choice set (created|completed|
  error) and --limit gets min=1, matching transcripts list.
- transcripts subcommands reordered so 'list' precedes 'get' in --help,
  matching sessions. (Bare 'assembly transcripts'/'assembly sessions'
  already rendered help via no_args_is_help.)
- Regenerated help snapshots; added mutation-killing tests for both branches
  of every new conditional.

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
- AMS session commands (balance/usage/limits/keys/audit/sessions) now carry a
  session-specific suggestion: a browser login is the only fix, so the dead-end
  "set ASSEMBLYAI_API_KEY" advice is dropped from resolve_session's error.
- resolve_api_key strips env/keyring/flag values and treats whitespace-only
  keys as missing, so ASSEMBLYAI_API_KEY='   ' hits the clean exit-4
  not-signed-in path instead of leaking a raw httpx illegal-header error.
- assembly login warns "Could not open a browser" when webbrowser.open returns
  False (headless Linux), not only when it raises — no more silent 120s wait.
- assembly login (both browser and --api-key paths) preflights
  config.keyring_usable() and fails fast (exit 2, keyring_unusable) with
  doctor's ASSEMBLYAI_API_KEY guidance before opening any browser.
- assembly whoami still renders the local profile/env/masked-key/session table
  when key validation fails on a network error; status reads "unreachable
  (network error)" (reachable: null in JSON), distinct from "key rejected",
  and exits 1 with a suggestion instead of swallowing the table.
- assembly usage validates --end >= --start and --window in {day, week, month}
  client-side (exit 2, before session/network); assembly audit --limit gets
  min=1 so zero/negative values are a click usage error.
- assembly logout stays idempotent (exit 0) but reports truthfully: "No stored
  credentials for '<profile>' — nothing to clear" on a fresh machine, and the
  JSON output now includes "cleared": true/false.
- --profile names are validated at resolution time (AppState.resolve_profile
  via the new public config.validate_profile), so a typo'd profile is a fast
  exit-2 in the root callback before any network round-trip.
- assembly keys create rejects empty/whitespace --name before auth/network.
- assembly login --json keeps stderr machine-readable: the flow's progress
  notes (opening browser / waiting / multi-org pick) ship as {"hint": ..,
  "url": ..} objects instead of prose, plumbed via run_login_flow(json_mode=).

Skipped: a `keys disable` subcommand — aai_cli/auth/ams.py has no AMS endpoint
wrapper for disabling tokens (only create/rename), so there is nothing to wire
a CLI command to. `keys` already had no_args_is_help=True; a test now pins it.

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
…essages

assembly eval:
- Per-row error handling: a failed transcription no longer aborts the run and
  discards completed (paid) rows. Failed rows are recorded (JSON "error" field,
  human ERROR column), the summary pools only the scored rows, and the command
  exits 1 with an eval_failed error. NotAuthenticated still aborts immediately
  (exit 4) so a rejected key doesn't bill through the dataset.
- Resolve the API key before downloading the dataset, so a signed-out user
  fails fast instead of pulling the whole dataset first.
- --collar without --speaker-labels is now a usage error (it was silently
  inert); the 1.0 default is applied internally.

eval_data:
- HF 401/403 errors surface the response body, and the HF_TOKEN hint only
  shows when the body reads like gating/auth — a proxy 403 ("Host not in
  allowlist") is no longer misattributed to a gated dataset.
- Manifest suffixes outside .csv/.jsonl are rejected with "Manifests must be
  .csv or .jsonl" instead of a confusing JSONL parse error.
- Grammar: "an audio column" / "a text column" (article matches the noun).

assembly transcribe:
- --json with a conflicting -o field (e.g. -o text) is now a usage error
  instead of -o silently winning over --json.
- A single local source with a non-audio extension (outside AUDIO_EXTENSIONS)
  gets a stderr warning (structured under --json, suppressed by --quiet).
- --audio-end now rejects negatives (min=0); --temperature is bounded to the
  SDK-documented 0..1 range.
- --out's parent directory is validated (exists + writable) before the billed
  transcription runs, exiting 2 up front instead of dying as internal_error
  afterwards; the '..' traversal check moved up front with it.
- Rejoined the docstring paragraph that rendered with a hard mid-sentence
  line break in --help.

youtube (yt-dlp wrapper):
- Download errors trim yt-dlp's report-a-bug boilerplate ("please report this
  issue on ... yt-dlp -U") and the redundant "ERROR:" prefix, keeping only the
  meaningful message.

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
…g hygiene

- speak: every --help example now puts the root --sandbox flag before the
  subcommand (the documented form errored with "No such option"), and the
  unsupported_environment error spells out the corrected invocation
  ("Re-run as: assembly --sandbox speak …").
- New aai_cli/streaming/diagnostics.py, shared by stream/agent/speak:
  silence_streaming_logging() extends the existing websockets silencing with
  the assemblyai.streaming SDK logger, so neither a reader-thread traceback
  ("unexpected internal error") nor the SDK's own "Connection failed: …"
  ERROR line duplicates the CLI's normalized error or pollutes --json stderr.
  stream_audio now calls it (agent/tts already silenced websockets via ws.py).
- WS handshake 401/403 now carry an actionable suggestion (whoami / key-env
  match / network-proxy access to the env's streaming host) in
  client.stream_audio, agent/session, and tts/session; 401 is
  NotAuthenticated (exit 4, rejected_key), 403 stays an APIError (exit 1).
- speak: --voice SPEAKER=VOICE mappings on non-speaker-labeled input now warn
  ("Ignoring --voice SPEAKER=VOICE mappings; input has no speaker labels.")
  instead of being dropped silently, mirroring the bare-voice warning; --voice
  help names example voices for discovery.
- agent/speak help docstrings rewrapped so Rich renders no widowed words.
- --sample-rate gains min=1 on stream and speak (the API requires a positive
  rate); stream's help now says it also declares the rate of raw PCM on stdin.
- agent --list-voices: the English/Multilingual grouping moved from comments
  into data (Voice dataclass with a language tag); the human list renders
  grouped and the --json array carries {name, language}.
- stream <downloadable URL>: regression test pinning that credentials resolve
  before yt-dlp runs (ordering was already correct at HEAD).

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
…s, telemetry disclosure

- Click parse errors (missing arg, bad option value) now honor --json: when the
  invocation opted into JSON, the error formatter emits the uniform
  {"error": {"type": "usage_error", ...}} envelope on stderr (exit code still 2)
  instead of the human Rich panel.
- Misplaced-flag guidance: unknown --json/-j at root level points at the
  per-command placement ("assembly <command> --json") instead of the misleading
  "(Possible options: --version)" guess; unknown -q/--quiet/--sandbox/--env/
  --profile/-p on a subcommand explains they are global flags that go before
  the subcommand.
- --sandbox with a disagreeing --env now exits 2 with a clear conflict error
  (human + JSON) instead of silently resolving to --env; the agreeing combo
  (--env sandbox000) still works.
- `assembly version` suggests `assembly --version` instead of "Did you mean
  'sessions'?"; dropped the stale "the version subcommand stays for parity"
  claim from the --version callback docstring.
- First-run telemetry disclosure: when the anonymous device_id is first minted,
  one stderr line discloses collection and the opt-outs; suppressed under
  --quiet/--json, printed at most once ever, and wrapped so it can never break
  a command. Adds the tiny additive config.has_device_id() probe.
- `telemetry status` now reports *why* (source: env:AAI_TELEMETRY_DISABLED /
  env:DO_NOT_TRACK / config / default) in both human and JSON output, and the
  hint flips to "Re-enable with 'assembly telemetry enable'." when disabled.
- doctor no longer claims "(HTTP 401)" for any rejected-key signal — the
  validate_key boundary can't distinguish 401 from 403/proxy blocks.
- Bare `assembly telemetry` shows help instead of "Missing command."
  (no_args_is_help; setup already had it).

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
…ard)

assembly init:
- An existing FILE target is now a clean usage error (exit 2,
  "<path> exists and is not a directory."), not an internal-bug traceback.
- --force preserves a configured (non-placeholder) .env key instead of
  silently resetting it to the placeholder, prints a stderr notice that
  existing files are being overwritten, and the --force help text notes
  the overlay (extra files are kept).
- The version banner prints only after validation passes, so pure error
  runs stay undecorated like the sibling commands.
- The report's `key` row is emitted symmetrically when a key IS found
  ("written — from environment/keyring"), including under --json.
- The --no-install sign-off hint (and the no-key launch-skipped row)
  carry the chosen --port into the `assembly dev` command.

assembly dev / assembly share:
- A busy --port substituting a neighbor is announced on stderr
  ("Port 5000 is in use; using 5001."), structured under --json and
  suppressed by --quiet (port 0 means "any", so no notice).
- share's tunnel-timeout error now names the cloudflared log file kept
  for debugging, and the temp log is deleted on clean exits.
- A failed webbrowser.open (headless box) prints a stderr hint with the
  URL instead of silence (covers init's launch path too).

assembly deploy:
- Gains the standard --json/-j flag: errors use the {"error": ...}
  envelope and the declined-confirmation path emits
  {"status": "aborted", "target": ...}.

assembly onboard:
- The wizard tracks per-section failures: any failed section ends the
  run with "Set up with N issue(s) (<sections> failed)." and exit 1
  (the auth stop keeps its exit 4) instead of a cheery exit 0.
- The environment section computes its summary from the actual checks:
  warnings-only runs say "Ready — 1 warning (only affects
  streaming/agent)." instead of "Everything looks good."
- --json now emits a machine-readable section summary on stdout (and
  forces the non-interactive prompter so prose can't corrupt it); a
  failing run also raises the standard JSON error envelope.

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
- Split test_init_command/test_login/test_onboard_sections to stay under
  the 500-line gate (new test_init_force_and_report, test_login_guards,
  test_onboard_environment modules; behavior unchanged).
- generated_code_compile_gate.py: capture stdout only (click CliRunner,
  mix_stderr=False) and disable telemetry, so stderr chrome like the new
  first-run telemetry notice can't corrupt the compiled fixtures; declare
  click as a dev dependency (DEP004-ignored like coverage).
- Reword two comment lines that tripped the net-new-Any counter.

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
Comment thread aai_cli/init/keys.py Outdated
@alexkroman
alexkroman enabled auto-merge (squash) June 12, 2026 01:23
claude added 3 commits June 12, 2026 01:46
…fix branch

Reconciles main's eval --concurrency thread pool with this branch's
per-row failure resilience: _transcribe_one returns per-item
Transcript-or-CLIError outcomes in both modes; NotAuthenticated and
non-CLIError bugs still abort and cancel queued items. DER breakdown
folded into _pooled_metrics; doctor keeps the no-status-code wording.

Post-merge fallout: eval_data's HF datasets-server client moved to
eval_hf_api.py and the eval concurrency/failure tests to
test_eval_failures.py (500-line gate); _transcripts split per mode
(xenon); init's key-source row no longer misreports a blank
ASSEMBLYAI_API_KEY as 'environment' (review finding).

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
…into the QA-fix branch

Main's #90 fixed two of the same QA findings differently: adopt its
--sandbox/--env conflict warning (--env wins, surfaced as {"warning": …}
in JSON mode) over this branch's hard usage error, and keep this
branch's fuller speak sandbox-hint suggestion which subsumes main's.

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
Five tests asserted plain substrings on Click usage errors; under
GITHUB_ACTIONS Rich forces color on and interleaves style codes
mid-message, so they failed only in CI. Assert on the color-free
render instead (the existing test_eval_command.py pattern).

https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
@alexkroman
alexkroman merged commit 908ddb5 into main Jun 12, 2026
8 checks passed
@alexkroman
alexkroman deleted the claude/youthful-cori-vb7ovs branch June 12, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants